• ModelBenchmarking

Goals

The fundamental goal of the Model Benchmarking phase is the re-calibration and benchmarking of knee models of each team, which would be the outcomes of Model Calibration, the previous modeling and simulation phase.

Specific goals of the Model Benchmarking phase are

• To re-calibrate two knee models relying on the workflow of the Model Calibration phase but using re-processed and explicitly described re-calibration data
  • one using Open Knee(s) data
  • one using Natural Knee data
• To benchmark two knee models
  • one using Open Knee(s) data
  • one using Natural Knee data
• To conduct simulations with models using
  • the same sample simulation case used in previous phases (passive flexion)
  • all loading cases of re-calibration data
  • all loading cases of earmarked benchmarking data

• To document and disseminate modeling & simulation (M&S) processes, including specifications and protocol deviations

• To document and disseminate load cases, parameters and results (errors) of re-calibration and benchmarking.
• To document and disseminate benchmarked models including changes in virtual anatomical and mechanical representations of model components.
• To document and disseminate simulation results.

Tentative Timeline

Earmarked Data

oks003 from Open Knee(s):

• Demographics
  • Left knee
  • Age: 25 years
  • Gender: Female
  • Height: 1.73 m
  • Weight: 68 kg
  • BMI: 22.8
• Specimen-specific mechanical testing and other relevant data sets for re-calibration

  • DESCRIPTION-DataRepresentation_OpenKnees.docx - This document provides a more explicit description of joint mechanics data including probed points, kinematics-kinetics conventions, and re-processing steps. READ THIS FIRST.

  • RECALIBRATION-Probed_Points_Files - Re-processed data for recalibration. Specifically, all probed points are provided in local bone coordinate systems.

  • RECALIBRATION-Processed_Data - Re-processed data for recalibration. Includes Includes passive flexion, and anterior-posterior translation, internal-external rotation, varus-valgus laxity at 0, 30, 60, and 90 degrees of flexion. Each direction of the laxity test are provided in separate files. Kinematics and kinetics data are provided in separate files. In csv (text) and png (image for visualization) formats. Kinematic offsets are also extracted and provided.

  • RECALIBRATION-PRE-Raw_Data.zip - Intermediate data, e.g. pre-processed to extract and specific laxity time-series data from raw files, including file format conversion, before extraction of re-calibration data time points. This is provided for completeness. The data set is not anticipated to be used.
• Specimen-specific mechanical testing and other relevant data sets for benchmarking

  • DESCRIPTION-DataRepresentation_OpenKnees.docx - This document provides a more explicit description of joint mechanics data including probed points, kinematics-kinetics conventions, and re-processing steps. READ THIS FIRST.

  • BENCHMARKING - Earmarked data for benchmarking. Includes combined 10 Nm valgus moment and 5 Nm internal rotation moment at 0, 30, 60, and 90 degrees of flexion. Kinematics and kinetics data are provided in separate files. In csv (text) and png (image for visualization) formats. Kinematic offsets are also extracted and provided.

  • BENCHMARKING-PRE-Processed_Data.zip - Intermediate data, e.g. pre-processed to extract specific time-points from combined loading time-series data, including file format conversion, before assembly of all re-calibration data time points. This is provided for completeness. The data set is not anticipated to be used.
  • BENCHMARKING-RAW.zip - Raw data, e.g. time-series data of original combined loading for different flexion angles, before file format conversion. This is provided for completeness. The data set is not anticipated to be used.
• Any other data disseminated in previous phases

• Download data in the downloads section of project site, https://simtk.org/projects/kneehub

  • Package name: Data for MB - oks003

  • Link to package location: https://simtk.org/frs/?group_id=1061

  • File name: data-MB-oks003.zip

  • Direct link to file: https://simtk.org/frs/download_confirm.php/file/6298/data-MB-oks003.zip?group_id=1061

DU02 from Natural Knee Data:

• Demographics
  • Right knee
  • Age: 44 years
  • Gender: Male
  • Height: 1.83 m
  • Weight: 70.31 kg
  • BMI: 21.02
• Specimen-specific mechanical testing and other relevant data sets for re-calibration

  • DESCRIPTION-NaturalKneeData_DataDescriptionStandard.docx - This document provides a more explicit description of joint mechanics data including probed points, kinematics-kinetics conventions, and re-processing steps. READ THIS FIRST.

  • RECALIBRATION-NaturalKneeData_StandardizedData.xlsx - Re-processed data for recalibration. Includes passive flexion, and anterior-posterior translation, internal-external rotation, varus-valgus laxity at four approximate flexion angles.

  • RECALIBRATION-PRE-NaturalKneeData_RawData.zip - Intermediate data, e.g. pre-processed for time synchronization, before extraction of re-calibration data points. This is provided for completeness. The data set is not anticipated to be used.

• Specimen-specific mechanical testing and other relevant data sets for benchmarking

  • DESCRIPTION-NaturalKneeData_DataDescriptionStandard.docx - This document provides a more explicit description of joint mechanics data including probed points, kinematics-kinetics conventions, and re-processing steps. READ THIS FIRST.

  • BENCHMARKING-NaturalKneeData_StandardizedData_Benchmark_v2.xlsx - Earmarked data for benchmarking. Includes passive flexion, and anterior-posterior translation, internal-external rotation, varus-valgus laxity at two approximate flexion angles, all following ACL resection.

• Any other data disseminated in previous phases

• Download data in the downloads section of project site, https://simtk.org/projects/kneehub

  • Package name: Data for MB - DU02

  • Link to package location: https://simtk.org/frs/?group_id=1061

  • File name: data-MB-DU02.zip

  • Direct link to file: https://simtk.org/frs/download_confirm.php/file/6301/data-MB-DU02.zip?group_id=1061

The modeling teams are expected to use all of the earmarked data, at least for simulations of all experimental load cases. They can also utilize information in literature to define model components that may not have any associated specimen-specific data.

Simulation Cases

Each team will simulate the same simulation case used in previous phases - passive flexion from 0 to 90 degrees, using the benchmarked models. The teams are expected to be faithful to their original interpretation of the term "passive flexion", and therefore its reflection as loading and boundary conditions. The simulations' primary aim will be the prediction of joint kinematics.

Each team will also simulate all the load case(s) of earmarked data for re-calibration and benchmarking even if they don't use the loading case for calibration. Execution of re-calibration and selection and interpretation of the load case(s) for re-calibration should be faithful to the workflows implemented by the individual teams during the Model Calibration phase. The simulation(s) will portray joint kinematics-kinetics response.

Deliverables

Modeling & Simulation Outputs

Intermediate and End-Point M&S Outputs

Two packages representing final and intermediate outcomes of Model Benchmarking phase will be the outputs of this phase. The focus is the complete curation of the processes and results of each individual team's Model Benchmarking activities.

• Package (*.zip) collating M&S Outputs using Open Knee(s)

  • End-point M&S Outputs such as

    • Working model including simulation cases in a format preferred by modeling team
    • Simulation results in a format preferred by modeling team

  • Intermediate M&S Outputs in formats preferred by modeling team

    • For re-calibration
      • Load case(s) used for re-calibration
      • Calibrated parameters
      • Calibration fit error
    • For benchmarking
      • Benchmarking error
    • Virtual representations of model components that are changed
    • For every load case (re-calibration or benchmarking)
      • Coordinate systems
      • Extracted joint kinematics-kinetics used for simulation
      • Loading and boundary conditions to represent loading case in simulations

• Package (*.zip) collating M&S Outputs using Natural Knee Data

  • Similar to package described above but utilizing Natural Knee Data

Documentation of M&S Outputs

Two document sets should provide information to help navigate the M&S output package and to describe what M&S Outputs are (not how they are obtained).

• Document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Outputs using Open Knee(s) data

  • Metadata for the document - authors, date, contact information
  • Information to navigate the package

  • Information to describe what each M&S Output is

• Document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Outputs using Natural Knee Data

  • Similar to document described above but utilizing Natural Knee Data

Modeling & Simulation Processes

An important set of deliverables are the documentation of the M&S processes, i.e. the individualized workflows to transform earmarked data to M&S outputs. The documents will include Specifications, to provide each team's initial plans for processing data to obtain M&S outputs, and Protocol Deviations, to describe modifications to Specifications during the execution of the initial plans. Ideally, when a third-party modeler has the earmarked data, specifications, and protocol deviations, they can follow the steps of the modeling team to reproduce their disseminated M&S outputs.

Specifications

Specifications documents are aimed to describe individual steps to obtain M&S outputs from earmarked data. The detail should be provided in a manner that other teams can reproduce the decision making process of the modeler. Any subjective and arbitrary decisions should be noted and as much insight as possible should be provided for justification and reproduction these decisions. This document should be prepared and disseminated before M&S activities start.

• Specifications document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Processes using Open Knee(s) data

  • Metadata for the document - authors, date, contact information

  • Information to describe what each target M&S Output is

  • Information to describe how each target M&S Output will be obtained

  • Anticipated burden of the workflow

• Specifications document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Processes using Natural Knee Data

  • Similar to document described above but utilizing Natural Knee Data

Protocol Deviations

If specifications need to be updated during activities to go from earmarked data to M&S outputs, these should be documented as protocol deviations. As much insight as possible should be provided for justification of the deviation. This document should be prepared during M&S activities and disseminated along with M&S outputs.

• Protocol Deviation document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Processes using Open Knee(s) data

  • Metadata for the document - authors, date, contact information
  • Collated set of protocol deviations, each describing
    • What is the deviation in relevance to Specifications?

    • Which M&S output does the deviation impact?

    • Who implemented the modification?
    • When was the modification made?
    • What is the justification for deviation?
  • Actual burden of the workflow

• Protocol Deviation document (in source, e.g., *.odt, *.docx, etc., and PDF formats) describing M&S Processes using Natural Knee Data

  • Similar to document described above but utilizing Natural Knee Data

Package for Comparative Analysis

Two packages will be prepared to curate M&S Outputs and process documentation of each team; one starting with data from Open Knee(s), the other utilizing Natural Knee data. The M&S Output files submitted by the individual teams will be organized in these packages and provided as is with the possible exception of renaming of master package files. The documents will be provided as is. Yet, additional organization of output description and process documentation may be necessary to meet third-party requirements or recommendations (see guidance from #USFDA, #ASME, #NASA, #Erdemir2015).

• Package of M&S outputs and processes using data from Open Knee(s)

  • Cleveland Clinic team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • University of Denver team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Cleveland State University team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Auckland Bioengineering Institute team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Hospital for Special Surgery team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

• Package of M&S outputs and processes using Natural Knee data

  • Cleveland Clinic team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • University of Denver team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Cleveland State University team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Auckland Bioengineering Institute team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

  • Hospital for Special Surgery team

    • M&S outputs package file

    • M&S outputs documentation files

    • M&S processes specifications documents

    • M&S processes protocol deviation documents

Guidance

Where to get the Earmarked Data?

The teams can download data in the downloads section of project site, https://simtk.org/projects/kneehub, check packages

• Data for Model Benchmarking - oks003, from Open Knee(s)
• Data for Model Benchmarking - DU02, from Natural Knee Data

Tell me more about re-calibration and benchmarking data from Natural Knee Data

DESCRIPTION-NaturalKneeData_DataDescriptionStandard.docx should provide clear guidance on data conventions, re-processing steps, and the content of final data files. Data sets that are bold in Earmarked Data (see above) are anticipated to be used. Those that are not bold are provided for completeness.

Known Limitations

This list provides identified limitations that needs to be managed by all teams while processing the data for modeling and simulation.

Clarifications

This list provides more information to clarify descriptions of data for modeling and simulation.

Tell me more about joint mechanics data from Open Knee(s)

DESCRIPTION-DataRepresentation_OpenKnees.docx should provide clear guidance on data conventions, re-processing steps, and the content of final data files. Data sets that are bold in Earmarked Data (see above) are anticipated to be used. Those that are not bold are provided for completeness.

Known Limitations

This list provides identified limitations that needs to be managed by all teams while processing the data for modeling and simulation.

Clarifications

This list provides more information to clarify descriptions of data for modeling and simulation.

How to prepare and submit Specifications?

Specifications are concerned about documentation of the processes to transform earmarked data to M&S outputs. This information is intended to be provided separately for processing of both data sets to capture nuances in response to variable data content and quality. This documentation will be provided before execution of Model Benchmarking activities to demonstrate the path each modeling team has chosen. Specifications should be in a detail such that other modeling teams can follow the steps to reproduce M&S outputs. When possible, justifications to support modeling & simulation decisions should be provided.

Recommended content for Specifications includes:

• Metadata for the document - contributors, contact information for corresponding author, date of document.

• Summary of data utilized with the M&S processes.

• Overview of target M&S outputs and M&S processes - similar to summarizing it in a scholarly publication.

• Detailed descriptions of target M&S outputs (consult Erdemir et al. (2012), for completeness)

  • Complete re-calibrated and benchmarked model
    • Anatomy
    • Mechanical properties
    • Coordinate systems
    • Loading and boundary conditions (for the simulation cases)
    • Interactions between model components (contacts, ties)
  • Intermediate and final outcomes of the phase
    • Re-calibrated parameters
    • Calibration fit error
    • Benchmarking error
    • Changed model components (anatomical and/or mechanical)
  • Intermediate and final outcomes of analysis of experimental load cases
    • Source data (as extracted from earmarked data set)
    • Processed data (as analyzed to make ready for use in simulations)
  • Simulation cases
    • Loading and boundary conditions
    • Target metrics for predictions
    • Numerical analysis settings
    • Anticipated results

• Detailed descriptions of M&S processes to get the target M&S outputs

  • Steps to re-calibrate and benchmark models
  • Steps to implement load cases as loading and boundary conditions
  • Steps to perform simulations
  • Burden
    • Software requirements
    • Hardware requirements
    • Anticipated man hours and expertise level
    • Computational cost
• References

Each team will submit two sets of documents, one for Model Benchmarking using Open Knee(s) data and another for Model Benchmarking using Natural Knee data. Each set will include the source document, i.e., in preferred format that it is written, and as a PDF copy. The documents should be submitted by any convenient means - e-mail, file sharing, upload to project site, etc. Ahmet Erdemir, erdemira@ccf.org, will push the submitted documents to the project site in the Documents section with labeling indicative of modeling phase, modeling teams, and data source.

How to prepare and submit Protocol Deviations?

It is likely that initial specifications may need to be modified to accomplish Model Benchmarking activities. These modifications, i.e. Protocol Deviations, should be documented during Model Benchmarking activities. Each protocol deviation should include information to understand who did, what and when, and why. The documentation should be in a detail such that other modeling teams can utilize it along with specifications to reproduce M&S outputs. This document should also include the actual burden of M&S activities including changes in software and hardware requirements, and deviations from anticipated labor effort and computational cost. Protocol Deviations should be collated in two sets of documents (one for using Open Knee(s) data and another for Natural Knee data). The documents will be provided after execution of Model Benchmarking activities in source form, i.e., in preferred format that it is written, and as a PDF copy. The documents should be submitted by any convenient means - e-mail, file sharing, upload to project site, etc. Ahmet Erdemir, erdemira@ccf.org, will push the submitted documents to the project site in the Documents section with labeling indicative of modeling phase, modeling teams, and data source.

How to curate, document and submit Modeling and Simulation Outputs?

All M&S outputs (intermediate and final) should be provided in their preferred format, i.e., accommodating individual teams' workflows. Each team should provide two packages (one for Open Knee(s) data and another for Natural Knee data) as a compressed archive (e.g. *.zip). For each package, an accompanying document should describe M&S outputs - in source form, i.e., in preferred format that it is written, and as a PDF copy. This information should focus on i) file descriptors and formats, and folder organization to help navigate the provided content and ii) what actually each M&S output is (NOT how they are obtained). The latter should be reflective of the description of target intermediate and final M&S products of Specifications. It should also be indicative of necessary information for other modeling teams to judge the nature and extent of the provided content. The packages should be submitted by any convenient means - e-mail, file sharing, upload to project site, etc. Ahmet Erdemir, erdemira@ccf.org, will push the submitted packages to the project site in the Downloads section with labeling indicative of modeling phase, modeling teams, and data source.

Preparing a package for Comparative Analysis

All material will likely need to be reorganized and converted to respond to third-party requirements for comparative analysis (see guidance from #USFDA, #ASME, #NASA, #Erdemir2015). Nonetheless, this should not prevent individual M&S teams to personalize their workflows and documentation style. Preparation of a package for comparative analysis will collate M&S outputs and documents submitted by the individual teams. M&S outputs will be provided in formats as delivered by the teams. Documentation may need to be reorganized using source copies of documentation from individual teams. Ahmet Erdemir, erdemira@ccf.org, will submit the prepared package to the third-party by any convenient means - e-mail, file sharing, upload to project site, etc.

References

Erdemir, A., Mulugeta, L. and Lytton, W. W. Ten “not so” simple rules for credible practice of modeling and simulation in healthcare: a multidisciplinary committee perspective, 2015 Biomedical Engineering Society / Food and Drug Administration Frontiers in Medical Devices Conference: Innovations in Modeling and Simulation, May 18-20, 2015, Washington, DC. Available at: https://simtk.org/svn/cpms/doc/posters/fmd_2015.pdf.

Erdemir, A., Guess, T. M., Halloran, J. P., Tadepalli, S. C. and Morrison, T. M. (2012) Considerations for reporting finite element analysis studies in biomechanics, Journal of Biomechanics, 45, 625-633. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3278509/.

U.S. Food and Drug Administration. Reporting of Computational Modeling Studies in Medical Device Submissions. Available at: https://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM381813.

NASA-STD-7009: Standard for Models and Simulations. Available at: https://standards.nasa.gov/standard/nasa/nasa-std-7009.

ASME V&V American Society of Mechanical Engineers - Committee Pages - V&V 40 Verification and Validation in Computational Modeling of Medical Devices. Available at: https://cstools.asme.org/csconnect/CommitteePages.cfm?Committee=100108782.